Typed Arrays from String Arrays for Dataset Operation

Preamble¶

Introduction¶

In this section, we're going to move from a raw dataset stored in a single string array (ndarray::Array2<String>) to multiple arrays of the desired type. This will enable us to use the appropriate operations for our different types of data. We will demonstrate our approach using the Iris Flower dataset.

Loading our Dataset¶

Before we move onto moving parts of our Iris Flower dataset into different typed arrays, we need to load it into our raw string array.

Let's display some rows from the string array to see if it;s loaded as expected.

Without digging deeper, it looks like we have the correct number of columns and rows.

Raw Dataset Dimensions¶

Once the data is loaded we may want to determine the number of samples (rows) and features (columns) in our dataset. We can get this information using the shape() function.

We can see that it's returned an array, where the first element indicates the number of rows and the second element indicates the number of columns. If you have prior knowledge of the dataset then this may be a good indicator as to whether your dataset has loaded correctly. This information will be useful later when we're initialising a new array.

Deciding on Data Types¶

Our dataset is currently loaded into a homogeneous array of strings, ndarray::Array2<String>. This data type has allowed us to load all our data in from a CSV file without prior knowledge of the suitable data types. However, it now means that we cannot apply pertinent operations depending on the feature. For example, we aren't able to easily determine any central tendencies for the SepealLenghCm column, or convert our units from centimetres to something else. All we can do right now is operate on these values as strings.

Let's see what happens if we try to operate on the data in its current form, e.g. if we want to find the mean average value of each column.

As expected, Rust is complaining about our data types. Let's look again at a summary of the dataset and make some decisions about our desired data types.

We can see that we have six columns, the names of which we have stored in our headers vector.

Let's go through them one-by-one and decide which data type will support our desired operations.

• Id. This is an identifier that came from the original CSV file. We don't have much use for this column in our upcoming analyses, so in this case, we're going to drop this column.
• SepalLengthCm, SepalWidthCm, PetalLengthCm, and PetalWidthCm. This is the multivariate data that describes each flower sample with regards to the length and width of the sepals and petals. These are numerical values with fractional parts, so we may want to store them in a floating-point data type, e.g. f32.
• Species. This column contains the true species of the flower samples. These are categorical values, so we may wish to convert them to numerical (integer) values, e.g. u32, or keep them as strings. We'll continue with the String type for now to keep things simple.

Moving Data to Typed Arrays¶

Once we've decided on what data types we want to employ we can move onto creating our typed arrays. This involves converting values from String to the desired type, and moving our data over to the new and typed arrays.

The Id Column¶

We've decided that we don't need this column, so it requires no action.

The SepalLengthCm, SepalWidthCm, PetalLengthCm, and PetalWidthCm Columns¶

We've decided that we want these columns to have a data type of f32, so we need to convert and move them into a new homogeneous array, this time of ndarray::Array2<f32>. We're going to achieve this by using std::str::FromStr which gives us access to the from_str() function that allows us to parse values from strings.

Let's demonstrate this approach on the first of the four columns, SepalLengthCm. We'll dump the column to an output cell to see the before and after.

It's clear from the output of data.column(1) that every element is a string. Now let's use mapv() to go through every element of data.column(1), parse each value to f32, and return the new array.

Looking at the output of our operations we can see that we were successful in parsing our string values into numerical ones. Let's now use this approach to create a new array named data_features of type ndarray::Array2<f32>. We'll need to convert our one-dimensional arrays into two-dimensional column arrays using insert_axis(Axis(1)) as we stack them.

If we don't want to copy and paste the same line of code multiple times (which we don't) we can use a loop instead. First we need to create an array of integers that identify the column indices of the features that we want to convert.

We can now iterate through the array of column indices, named selected_features, and stack each converted column into a new array of type Array2::<f32>.

Our headers vector (which describes 6 columns with 6 elements) doesn't broadcast onto our new features array (4 columns), so we'll create a new headers vector, feature_headers.

We now have our floating-point typed features (SepalLengthCm, SepalWidthCm, PetalLengthCm, and PetalWidthCm) in a single array. Let's see how the data looks in its new form.

We're only seeing 10 samples array summary, so let's plot our features to get a better idea.

Let's also check to see that we can now calculate the mean average per column.

It appears to be operating as expected. You could validate the results by running the same operation in different software.

The Species Column¶

We've decided to keep these as strings. Let's index the Species column and store it in its own array named labels.

Finally, we'll check the first 10 elements of our new labels array.

We could also use the itertools crate to apply some better presentation to this output.

Conclusion¶

In this section, we've demonstrated how to get parts of our raw string array into multiple arrays of various type. With this approach, we can now start operating on our data using appropriate operators for our analyses.