MLOps

There is a ‘T’ before the L in the ELT — the decision by the EL provider to create as highly normalized as possible landing tables.
This is beneficial to them because when priced on Monthly Active Rows it means more rows are filled, hence more revenue, and it creates more cloud credit attribution which can be used for increases in valuation.

https://plotly.github.io/next-static/

When you do analytics in a transactional (OLTP) database, life is full of tradeoffs. They’re great at acting on individual rows extremely quickly, but for large query loads like those required for reporting, row-based databases like MySQL are sloooow.
…
Among other differences, cloud data warehouses are columnar, meaning they store + query each column separately.
That means if you only select 2 columns from a table, the query will truly only fetch those two columns—as opposed to a row-based transactional database, which would first get entire rows, and then lop off all but those two columns.

tldr; ELT is just EtLT

You could call this the ETLT workflow (extract, normalizing transform, load, analytics transform), but most of us just drop the first T and refer to this as the ELT workflow (extract, load, transform).
These naming conventions do not need to be an exact science. What’s important is that there’s a final T at the end of “ELT workflow.” This finality implies freedom—the freedom of analytics teams to transform data to meet the needs of many downstream users (reporting, ML modeling, operational analytics, exploratory analysis, etc).
Without this T living at the end of the workflow, analytics teams are boxed in by the format the data arrives in.

https://hightouch.io/blog/reverse-etl/
ETL vs Reverse ETL seems like a false dilemma

Reverse ETL is the process of copying data from your central data warehouse to your operational tools, including but not limited to SaaS tools used for growth, marketing, sales, and support.

“I’ve spent all of this money and time getting my data into a warehouse to serve as my single source of truth, and now you’re telling me I need to take my data back out of the warehouse?”

https://hightouch.com/blog/reverse-etl-vs-elt

Without Reverse ETL, your data warehouse is a data silo. All of the data within it is only accessible to your technical users who know how to write SQL. The data only exists in a dashboard or report used to analyze past behavior and not drive future actions. Your business users who are running your day-to-day operations across sales, marketing, support, etc. need access to this data in the tools they use on a daily basis (i.e. Hubspot, Salesforce, Braze, Marketo, Amplitude, etc.).

to be ML productive at reasonable scale you should invest your time in your core problems (whatever that might be) and buy everything else.
Since we are dealing with reasonable scale, there is not much value in devoting resources to deploy and maintain functionalities that today can be found as PaaS/SaaS solutions (e.g. Snowflake, Metaflow, SageMaker).

yet, the idea of using open source tools is often frowned upon by team leaders and execs

a possibly larger AWS bill is often offset by higher retention rate and greater ML productivity.

recruiting and retention in a competitive market are constant challenges for companies, especially in ML
one of the main reasons for turnover of ML practitioners is devoting a sizable portion of their time to low-impact tasks, such as data preparation and infrastructure maintenance.

ML models at a RS create monetary gains for hundreds of thousands to tens of millions of USD per year (rather than hundreds of millions or billions).
ML in RS companies can have considerable impact, but the absolute scale of such impact rarely reaches the scale of big-data companies.

RS companies have dozens of engineers (rather than hundreds or thousands).

To optimize for smaller teams, RS companies are often striving to minimize operational friction, that is, to find ways for ML developers and data scientists to rely as little as possible on other teams to get data, provision GPUs, serve models, etc.

much of ML systems’ development depends on the type of problem solved, so data scientists need to be able to make choices about tooling, architecture and modeling depending on datasets, data types, algorithms and security constraints.
In addition, ML systems are not deployed against static environments, so data scientists need to be aware of changes in the data, changes in the model, adversarial attacks, and so on.
At the same time, it is important not to have data scientists involved in too many ancillary tasks, as it would require them to develop too many complementary skills: if it is now their job to provision GPUs, we simply shifted the burden, rather than increasing velocity. Striking the right balance is no mean feat.

RS companies deal with terabytes (rather than petabytes or exabytes).
For RS companies, instead, collecting massive training sets is typically not feasible due to issues such as data scarcity, privacy protection, and regulatory compliance or simply mere scale.⁴
Perhaps, excessive focus on top-notch modeling is less than strategically wise. For instance, best in class models might not be optimal in terms of cost/gain ratio or in many cases might not even be a viable option if they are too data hungry.

There is much more marginal gain for RS companies from focusing on clean, standardized and accessible data, as suggested by the original proponents of Data-Centric AI.

RS companies have a finite amount of computing budget.
One of the factors that impacts computational efficiency the most in RS companies is the inefficient design of ML systems from an end-to-end perspective (including the data stack). At RS the focus needs to be equally split between keeping the bill as low as possible and making scaling as efficient as possible.

The paradox is that it is easier for Google to go from 1 GPU to 1000 GPUs than it is to go from 1 GPU to 3 GPU for most RS companies. For instance, plenty of RS companies use distributed computing systems, like Spark, that are unlikely to be required. Much can be achieved with an efficient vertical design that encompasses ways to scale computational resources with minimal effort and only when needed.

The greater marginal gain for a RS company is always in having clean and accessible data: good data is more important than relative improvements in modeling and model architecture.

• training samples are inherently scarce and iterations cannot be super fast
  
• data ingestion must include getting data through a standard
  you can pick a domain-specific protocol that already exists or come up with your own standard if your data are somewhat unique to your business.
  
  • two events that are meant to describe the very same thing can never have different structures. For every exception to this rule there will be a price to pay sooner than later.
    
  • loose validation and rejection. In general, events should never be rejected, even when they don’t adhere to the standard format agreed upon. The system should flag all ill-formed events, send them down a different path and notify an alert system.
    

The first consequence of this principle is that data ingestion is a first-class citizen of your MLOps cycle and getting clear data should be the ultimate goal.

A clear separation between data ingestion and processing produces reliable and reproducible data pipelines. A data warehouse should contain immutable raw records of each state of the system at any given time.

• the data pipeline should always be built from raw events and implement a sharp separation between streaming and processing.
  
  • The role of the first is to guarantee the presence of truthful snapshots of any given state of the system,
    
  • the latter is the engine to prepare data for its final purpose.
    

The crucial distinction is that the output of the streaming is immutable, while the output of processing can always be undone and modified. The idea here is somewhat counterintuitive if you were raised with the idea that databases are about writing and transforming data, but it really is quite simple at its core: models can always be fixed, data cannot.

• your system allows for replayability?
  can you change something in your data transformation, in your queries or in your models and then replay all the data you ingested since the beginning of time without any major problem (time aside)?
  If the answer is no, you should really try to change that into a yes, and where not possible, strive to get as close as possible.
  

Focus is the essence of the RS. Instead of building and managing every component of a ML pipeline, adopt fully-managed services to run the computation.

when resources are limited, it is a good practice to invest them in the most important business problem.

fully-managed services tends to be more expensive in terms of hard COGS (costs), but at the same time data scientists can stop worrying about downtime, replication, auto-scaling and so on.
All those things are necessary, but that does not mean they are central to the purpose of your ML application and
in our experience, the benefits of keeping your organization uniquely focused on the core business problems very often outweigh the benefit of low bills (within reasonable limits)
Because maintaining and scaling infrastructure with dedicated people is still going to be expensive, so in the end it can easily end up being much more costly than paying and managing a handful of new providers.

Moreover, the worst thing about building and maintaining infrastructure is that the actual costs are rather unpredictable over time. Not only is it extremely easy to underestimate the total effort required in the long run, but every time you create a team for the sole purpose of building a piece of infrastructure you introduce the quintessential unpredictability of the human factor.

Consumption bills have very few positive aspects, but one they have for sure is that they can be easily predicted

invest your time in your core problems and buy everything else

A RS company does not require distributed computing at every step. Much can be achieved with an efficient vertical design.

• Distributed systems, like Hadoop and Spark, played a pivotal role in the big-data revolution.
  Only five years ago Spark was pretty much the only option on the table to do ML at scale.
  Startups all around the world use Spark for streaming, SparkSQL for data exploration and MLlib for feature selection and building ML pipelines.
  However, they are cumbersome to work with, hard to debug and they also force programming patterns unfamiliar to many scientists, with very negative impacts on the ramp up time of new hires.
  

The point that we want to make is simple: if you are a RS company, the amount of data you deal with does not require ubiquitous distributed computing: with a good vertical design it is possible to do much of the work while improving significantly the developer experience.

• your ML pipeline should be conceived as a DAG implemented throughout a number of modular steps
  while some of them might require more computational firepower, it is important to abstract away the computation piece from the process and the experience of ML development.
  
• Distributed computing should be used for solving vast and intricate data problems.
  For everything else, it is more practical to run the steps in the pipeline in separate boxes, scaling up computation in your cloud infrastructure at and only at the steps that require it, like Metaflow allows to do for example.
  
• this shift in perspective has a massive impact on the developer experience of any ML team, because it will give data scientists the feeling of developing locally, without having to go through the hoops and hurdles of dealing with distributed computing all the time.
  

• We want to encourage vertical independence of ML teams.
  
  • Much of the work in ML depends heavily on the type of problem solved, so:
    
  • data scientists need to be able to make reasonably independent choices about tooling, architecture and modeling depending on datasets, data types, algorithms and security constraints.
    
  • ML systems are not deployed against static environments, so
    
  • data scientists need to be aware of changes in the data, changes in the model, adversarial attacks, and so on.
    

• do not destroy the past forever
  
  • we can’t “time-travel” anymore in our data stack: we can’t debug errors from last week in data-driven system whose input is the catalog — data would have changed by now, and reproducing the state of the system is impossible;
    
  • by losing track of the original state, any update or modification is much harder to undo, in case of mistakes in the business logic;
    
  • it becomes harder to test new code and iterate on it, as different runs will produce slightly different outcomes.
    
• append-only log pattern
  a never-ending log table and then practical snapshots, capturing the state of the universe at certain points in time
  
  • maintaining a write-only table that stores all data relevant to the universe in which we operate, in one immutable ledger
    
  • since we cannot change the past, our log-like table will be a perpetual memory of the state of our universe at any given point in time.
    
  • From this representation, it is always possible to recover the “previous table” as a view over this stream, that is, a “photograph” capturing for each product the freshest info we have.
    
  • having the freshest table does not destroy previous information, in exactly the same way as running the latest code pulled from git does not destroy previous versions of it
    
• This is now way easier to implement than before, as a result of three main trends:
  
  1. data storage is getting cheaper by the minute,
     
  2. modern data warehouses have superb query performances and great JSON support,
     
  3. open source tools like dbt made it easier for people with a broad range of skills to transform data in a versioned, replayable, testable way.
     

The ingestion pipeline mimics a typical data flow for data-driven applications: clients send events, an endpoint collects them and dumps them into a stream, finally a data warehouse stores them for further processing.

In our example, we are providing recommendations for myshop.com, so:

• The clients sending events are the shoppers browsers: as users browse products on myshop.com, a Javascript SDK sends analytics to our endpoint to collect behavioral signals (to be later used for our recommender!). To simulate these on-site events, the repository contains a pumper script, which streams realistic payloads to the cloud;
  
• the endpoint is responding with a pixel, a 1x1 transparent gif used for dealing with client-server tracking (e.g. Google Analytics);
  
• behavioral signals are dumped into a raw table in Snowflake;
  
• SQL-powered transformations (through dbt) take the raw events and build features, intermediate aggregations, normalized views etc. While detailing downstream applications is outside the scope of this post (but see this for a fleshed out example), you can imagine B.I. tools and ML pipelines to use these nicely prepared tables as input to their own workflow.
  

The endpoint runs in a serverless fashion and will scale automatically thanks to AWS lambda (same goes for data stream on Firehose); Snowflake computing power can be, if necessary, adjusted per query, without any maintenance or arcane configuration (Spark, thinking of you!); the dbt flow can be run either in a SaaS orchestrator (e.g. Prefect) or, even through the “native” dbt cloud.

Once again, our proposal sits in the middle of the spectrum, between the custom infrastructure of tech giants running “impossible scale” loads, and monolithic, end-to-end enterprise platforms, that offer little-to-no-control over the available functionalities.

In the Big Data era, lambda architectures were a popular solution for data-driven applications where both historical data — aggregations and counts for the last 30 days — and fresh data — aggregation and counts for the last hour — matter, for visualization, monitoring or ML purposes.
In a nutshell, you run a batch and a streaming system in parallel and join results for clients downstream.

Lambdas faded a bit:

• many critics advocated for leaner solutions not involving maintaining de facto two stacks, and,
  
• much ML still today doesn’t really use real-time data — and even when it does, it is often the “online prediction with batch features” scenario.
  

Can we dream again the lambda dream and leverage the modern data stack to get the cake and eat it too?

The intuition is that the ever-growing log table (green) can logically be split into two parts: the distant past (orange), immutable and for which sessions have been already assigned, once and for all, and the immediate past (pink), where new events are still coming in and things are still in flux. We can then run the same SQL logic in one stack, avoiding therefore two big drawbacks of original lambda architectures
Barbell strategy applied to distributed systems, means a (false) tradeoff between consistency and availability
While this pattern does not replace a full-fledged low-latency streaming solution, it opens the possibility of doing almost real-time (~5 min) analysis on terabytes of data at a scale and speed that the original lambda architecture, with all its complexity, could only dream of.

If you have already adopted Snowlake, dbt and the log pattern, it also helps you bridge the gap between batch and real-time in a gentle way, before investing in a parallel stack for streaming that you may not need if your acceptable latency is measured in minutes.

The first step is to enable data science and machine learning practitioners to use K8s clusters as pools of compute resources. Data science is an inherently compute-heavy activity and we can use K8s as an engine to power it.
Metaflow makes it easy to start by running everything locally but when you need to scale, it allows you to smoothly move parts of your workflows to K8s by adding a single line of code.
«Literally is just using a decorator =@kubernetes=»

Esto lleva a que se empieza a coger datos de bases de datos que no estaban pensados para ellos y a hacer ingeniería inversa la lógica de negocio
Cuando se rompe algo en producción no se sabe de qué equipo es la culpa

ML Data Scientists operate in two primary mindsets: Analysis and Creation.

• Analysis is:
  
  • improving their understanding of the context (business, environment, data, product).
    
  • updating their mental model of the world (queries, stats, diagrams).
    
  • shortlived and dynamic, emphasis is on speed.
    
• Creation is
  
  • actively influencing the world through building ML products (code, data transformations, datasets, models). Of course, next, they switch back to analysis mode to understand if they are influencing in the right way.
    
  • long-term, iterated, emphasising agility (ability to respond to change).
    

SWE: if something is wrong and breaks specs, you can immediately know
ML: You can only validate your work over a longer term.

The last two points are mostly copy paste

• Intro to Data Build Tool (dbt) // Create your first project! - YouTube
  

pip install dbt-postgres
mkdir dbt_test && cd dbt_test
dbt init

• config is stored in ~/.dbt/profiles.yml, with one configuration per environment, edit it
  

dbt debug # test connection
dbt run # run models

• dbt seed allows you to copy csvs
  
• dbt compile to get the actual sql migrations
  

—

• dbt models
  A model is a select statement. Models are defined in .sql files (typically in your models directory):
  

  • Each .sql file contains one model / select statement
    
  • The name of the file is used as the model name
    
  • Models can be nested in subdirectories within the models directory
    

  When you execute the dbt run command, dbt will build this model in your data warehouse by wrapping it in a create view as or create table as statement.
  

  • https://www.startdataengineering.com/post/dbt-data-build-tool-tutorial/
    
    • can depend on other models
      
    • have tests defined on them (schema test and data test)
      
    • can be created as tables or views (materializations, which are strategies for persisting dbt models in a warehouse)
      
• jinja & macros
  Using Jinja turns your dbt project into a programming environment for SQL, giving you the ability to do things that aren’t normally possible in SQL. For example, with Jinja you can:
  
• Use control structures (e.g. if statements and for loops) in SQL
  
• Use environment variables in your dbt project for production deployments
  
• Change the way your project builds based on the current target.
  
• Operate on the results of one query to generate another query, for example:
  
  • Return a list of payment methods, in order to create a subtotal column per payment method (pivot)
    
  • Return a list of columns in two relations, and select them in the same order to make it easier to union them together
    
• Abstract snippets of SQL into reusable macros — these are analogous to functions in most programming languages.
  

The “Airflow + Dbt examples” is mostly unprocessed

they both provide common interfaces that data teams can use to get on the same page.

• Airflow provides common interface to logging storage, view, monitoring, and rerunning failed jobs (overall pipeline health and management) → pipeline orchestation
  
• dbt provides common interface to boilerplate DDL, dependency management between scripts, … → data transformation layer
  

https://www.reddit.com/r/dataengineering/comments/ve2qbp/why_does_dbt_have_so_much_hype_metions_in_this/
Useful for people without software engineering to write clean code, easy to learn

• The use of Jinja macros with SQL
  
• The ability to see lineage
  
• The ability to automate the generation of docs for your users
  
• That you don’t need to worry about the order of table updates
  
• That you can very simply add tests to your code in a standard and repeatable fashion.
  

https://www.reddit.com/r/Python/comments/pwq8q7/etl_library_for_python/

pandas is good.
pyjanitor can extend pandas and help you write clean code to clean data.
dask can help pandas scale.
prefect can orchestrate.
airflow can also orchestrate.
pyspark scales but uses spark under th hood.
koalas is like pandas, but spark under the hood.
great_expectations can test and document data.
pandera can statistically test data.

Also does orchestration

https://mattturck.com/data2021/

https://www.reddit.com/r/dataengineering/comments/oyju56/dataengineering_2021_in_one_pic/?utm_medium=android_app&utm_source=share

https://cnvrg.io/mlcon/

The key principles of this concept are:

• Individuals and Interactions: Business intelligence is driven by what users ask about their business. The technical setting is secondary.
  
• Business Driven: Well documented data warehouses that take years to deploy will always be out of date. Business users will look elsewhere. My experiences with business units: I need it now or I’d rather stick with Excel solution ….
  
• Customer Collaboration: End users’ knowledge of their business is your greatest resource.
  
• Responding to Change: If you take all of the above actions, change will come naturally and result in weekly delivery cycles.