AWS Glue DataBrew, how does it compare to the AI & Analytics Engine’s Smart Data Preparation? 

Similarities

The main similarity between the two tools is that they both cater to the need for an easy-to-use no-code interactive data preparation tool. Let's take a closer look: 

Data Ingestion and Outcome

Both tools are targeted at tabular data files in the following formats: CSV, JSON (lines), and PARQUET.

While DataBrew allows the import of data mainly from data lakes on the AWS cloud such as S3, Redshift, and RDS — The Smart Data Preparation (AI & Analytics Engine) allows a diverse set of options for importing data, such as HTTP (URLs), SQL and NoSQL databases. Users can still upload data from cloud storage services such as S3 or GCS (google cloud) by getting a pre-signed URL for their dataset and using the HTTP option.

The outcome of the recipe-building process is a re-runnable recipe that can be re-used to transform a larger dataset of the same input schema.

Interactive Recipe Building and Recommender

Both AWS Glue DataBrew and the Smart Data Preparation feature have an interactive recipe-building user interface. There are many similarities between the two:

1. Quick preview of the dataset being prepared
2. The list of actions (steps) selected so far in the recipe
3. The ability to edit or delete a step in the recipe

Validation of Recipe

Whenever the list of actions in a recipe is modified, a validation check is applied to ensure that the recipe is legitimate. It consists of the following among many checks:

• • Input columns are available in the schema and are of the correct data types
  • The combination of parameters is valid
  • Names of new columns output by an action are not colliding with existing column names

Advanced column selectors

Our Smart Data Preparation offers the ability for users to apply a transformation on multiple columns with a single action. To aid these, our graphical user interface and API provides the following modes of including/excluding input columns, column groups, or both into the selection:

• • By Name: explicitly list out the name
  • By Type: select by schema type
  • By Pattern: select by match in regex pattern of the name

The user is also allowed to combine multiple such criteria with the and/or operator. This enables full flexibility to let the user specify complex selection criteria such as “columns matching the name pattern ‘x_.*’, excluding non-numeric columns.”

Queuing and Committing actions to a recipe

In The AI & Analytics Engine, whenever an action is added, it is “queued” to be committed to the recipe. Like in DataBrew, the queued actions are run on a fixed-size sample (first 5k rows) of the full dataset and the result is displayed as a preview. This serves as visual feedback to users, allowing them to change the configuration of their action to ensure that the preview represents what they desire as the result.

Our platform also provides an additional functionality enabled for the user, called the “committing” of the queued actions to a recipe, before continuing to edit the recipe further. Committing of actions signals the platform to run the recipe actions on the full dataset (rather than on the sample) and then show a renewed sample preview. This results in a more accurate data preview, where the sample was first generated from the raw data before the recipe actions were applied.

Committing actions to a recipe also provides the platform to run intelligent algorithms with the fully processed data to provide users with good recommendations for the next set of actions in their recipe.

Recommendations

In DataBrew, recommendations are generated on a per-column basis, and are not available by default. The user needs to click on a particular column and request recommendations corresponding to one column.

The AI & Analytics Engine’s Smart Data Preparation also provides recommendations of the next set of actions likely to be helpful to the user. 

The key differences are that: 

1. These recommendations are generated automatically the first time the user starts a recipe and after every commit. 
   
   
2. The recommendations are based on a sample of the entire dataset rather than a single column. This makes it quite attractive for users who want to detect, for example, input columns with too little correlation with the target variable that needs to be dropped.
   
   
3. Every recommendation is accompanied by reasons why these actions are useful, showing charts and summary statistics to help the user understand and scrutinize their data. This is particularly useful for users without data science expertise who want to access the benefits of AI/ML.