Pipeline Architecture AKA Pipes and Filters

Spread the love

  • sometimes is used as a pattern in solutions
  • the data flow in pipes between filters
  • he data flows unidirectionally
  • the paradigm is identical to the piping in Linux “ls | grep data
  • there are 4 types of filters (ways of handling data)
  • producer – the origin of data (source)
  • transformer – receives data and can transform it (map)
  • tester – tests the data and outputs some of it (reduce)
  • consumer – the data flow stops (destination)
  • examples: ETL tools, orchestrators, document transformations, operating systems
  • pros: Simple, testable, Modular
  • cons: scalability, fault tolerance, elasticity, deployability, performance

Leave a Reply

Your email address will not be published. Required fields are marked *