We also built new tooling for executing data quality checks and anomaly detection, and required their use in new pipelin

In our case, we implemented classical MVC (Model — View — Controller) architecture with a public API instead of View and dedicated Interactors, which encapsulate all business logic related to each endpoint.

GetNewsInteractor is using Codeforces, YouTube, and other repositories to fetch data from 3rd party resources and database, which is abstracted through Exposed library (officially supported by JetBrains as well).

Voilà! We have our own fully-functional backend written in Kotlin, which is perfectly suited to our needs, and open to changes in any direction we would like to take in the future. And we still can use some Firebase on the backend ;)

Some functions require a long list of arguments. Although this should be avoided altogether (e.g. by using data classes), it’s not always up to you. In such cases, the second-best option is to create a dictionary with all the named arguments and pass that to the function instead. It will generally make your code more readable.

As we set out to rebuild our data warehouse, it was clear that we needed a mechanism to ensure cohesion between data models and maintain a high quality bar across teams. We also needed a better way to surface our most trustworthy datasets to end users. To accomplish this, we launched the Midas certification process (depicted in the diagram below).

Pipelines are used in Ktor as an extension mechanism to plug functionality in at the right place. For example, a Ktor application defines five main phases: Setup, Monitoring, Features, Call and Fallback. The routing feature defines its own nested pipeline inside the application’s call phase.

Last, but not least, we created new mechanisms for ensuring accountability related to data quality. We refreshed our process for reporting data quality bugs, and created a weekly Bug Review meeting for discussing high priority bugs and aligning on corrective actions. We also require that teams incorporate data pipeline SLAs into their quarterly OKR planning.

As a company matures, the requirements for its data warehouse change significantly. To meet these changing needs at Airbnb, we successfully reconstructed the data warehouse and revitalized the data engineering community. This was done as part of a company wide Data Quality initiative.

http://vert.actiup.com/eil/Video-torino-v-verona-v-yt2-1alp-14.php

http://m.dentisalut.com/qtk/videos-Lazio-Fiorentina-v-en-gb-1jos-16.php

http://old.cocir.org/media/los/v-ideos-denizlispor-v-kaiserispor-v-yt2-1hug-28.php

http://stream88.colomboserboli.com/eca/v-ideos-lill-v-anzher-v-yt2-1thk-9.php

http://skrs.vidrio.org/sbe/videos-sassuolo-v-dzhenoa-v-yt2-1kla-21.php

http://agro.ruicasa.com/kjv/video-Enyimba-Al-Merrikh-v-en-gb-yui30122020-.php

http://m.dentisalut.com/qtk/videos-Lazio-Fiorentina-v-en-gb-1zqh-25.php

http://vert.actiup.com/eil/video-torino-v-verona-v-yt2-1gea-8.php

http://stream88.colomboserboli.com/eca/video-sent-eten-v-pszh-v-yt2-1lzb-1.php

http://old.cocir.org/media/los/Video-denizlispor-v-kaiserispor-v-yt2-1cul-19.php

http://skrs.vidrio.org/sbe/videos-napoli-v-spetsiia-v-yt2-1bva-11.php

http://stream88.colomboserboli.com/eca/videos-sent-eten-v-pszh-v-yt2-1cza-29.php

http://old.cocir.org/media/los/video-denizlispor-v-kaiserispor-v-yt2-1gxp-11.php

http://skrs.vidrio.org/sbe/v-ideos-napoli-v-spetsiia-v-yt2-1aun-10.php

http://vert.actiup.com/eil/Video-torino-v-verona-v-yt2-1exo-15.php

http://m.dentisalut.com/qtk/Video-lazio-v-fiorentina-v-it-it-1wrm2-11.php

http://skrs.vidrio.org/sbe/v-ideos-napoli-v-spetsiia-v-yt2-1ieb-27.php

http://m.dentisalut.com/qtk/videos-lazio-v-fiorentina-v-it-it-1qfv2-5.php

http://agro.ruicasa.com/kjv/v-ideos-vita-club-v-young-buffalos-fc-v-fr-fr-1hud-27.php

http://stream88.colomboserboli.com/eca/v-ideos-sent-eten-v-pszh-v-yt2-1unk-10.php

made it safely to the end of the round, you’d win and all the other lemmings would follow. If the lead lemming fell off a cliff into a pit of fire, so did the rest of his crew, even though they saw him fall in first they kept going.

That’s nice, but you probably already knew these things. What you might not know is that you can also force keyword arguments. The details are described in PEP 3202, but it comes down to using an asterisk before the arguments you want to force as keyword arguments. Or, before everything, forcing all arguments to be keyword arguments:

The Midas process requires stakeholders to first align on design specifications before building their pipelines. This is done via a Spec document that provides layman’s descriptions for metrics and dimensions, table schemas, pipeline diagrams, and describes non-obvious business logic and other assumptions. Once the spec is approved, a data engineer then builds the datasets and pipelines based on the agreed upon specification. The resulting data and code is then reviewed, and ultimately granted certification. The certification flags are made visible in all consumer facing data tools, and certified data is prioritized in data discoverability tools.

To create Ktor server, you simply need to create new project, declare the application entry point, and install a few features to start with. In our case, these features are ContentNegotiation and Routing.

Decorators are wrappers around a function that modify the behavior of the function in a certain way. There are many use cases for decorators, and you may have used them before when working with frameworks like Flask.

The Data Quality initiative accomplished this revitalization through an all-in approach that addressed problems at every level. This included bringing back the Data Engineering function, setting a high technical bar for the role, and building a community for this engineering specialty. A new team was also formed to develop data engineering-specific tools. The company also developed a highly opinionated architecture and technical standards, and launched the Midas certification process to ensure all new data was built to this standard. And finally, the company up-leveled accountability by setting high expectations for data pipeline owners, specifically for operations and bug resolution.

Data operations was another opportunity for improvement, so we made sure to set strict requirements in this area. All important datasets are required to have an SLA for landing times, and pipelines are required to be configured with Pager Duty.