When we began the Data Quality initiative, most critical data at Airbnb was composed via SQL and executed via Hive. This

As a company matures, the requirements for its data warehouse change significantly. To meet these changing needs at Airbnb, we successfully reconstructed the data warehouse and revitalized the data engineering community. This was done as part of a company wide Data Quality initiative.

GetNewsInteractor is using Codeforces, YouTube, and other repositories to fetch data from 3rd party resources and database, which is abstracted through Exposed library (officially supported by JetBrains as well).

Some functions require a long list of arguments. Although this should be avoided altogether (e.g. by using data classes), it’s not always up to you. In such cases, the second-best option is to create a dictionary with all the named arguments and pass that to the function instead. It will generally make your code more readable.

The Midas process requires stakeholders to first align on design specifications before building their pipelines. This is done via a Spec document that provides layman’s descriptions for metrics and dimensions, table schemas, pipeline diagrams, and describes non-obvious business logic and other assumptions. Once the spec is approved, a data engineer then builds the datasets and pipelines based on the agreed upon specification. The resulting data and code is then reviewed, and ultimately granted certification. The certification flags are made visible in all consumer facing data tools, and certified data is prioritized in data discoverability tools.

The Data Quality initiative accomplished this revitalization through an all-in approach that addressed problems at every level. This included bringing back the Data Engineering function, setting a high technical bar for the role, and building a community for this engineering specialty. A new team was also formed to develop data engineering-specific tools. The company also developed a highly opinionated architecture and technical standards, and launched the Midas certification process to ensure all new data was built to this standard. And finally, the company up-leveled accountability by setting high expectations for data pipeline owners, specifically for operations and bug resolution.

Voilà! We have our own fully-functional backend written in Kotlin, which is perfectly suited to our needs, and open to changes in any direction we would like to take in the future. And we still can use some Firebase on the backend ;)

http://live-stream.munich.es/twr/Video-fenerbahce-v-alanyaspor-v-tr-tr-1kei-2.php

http://live07.colomboserboli.com/niy/v-ideos-Fenerbahce-Alanyaspor-v-en-gb-1ptd-.php

http://go.acaps.cat/vac/v-ideos-fenerbahce-v-alanyaspor-v-tr-tr-1egp-9.php

http://live-stream.munich.es/twr/v-ideos-fenerbahce-v-alanyaspor-v-tr-tr-1hiq-12.php

http://news24.gruposio.es/ktn/v-ideos-fenerbahce-v-alanyaspor-v-tr-tr-1vcw-22.php

http://live07.colomboserboli.com/niy/videos-Fenerbahce-Alanyaspor-v-en-gb-1yqb-15.php

http://news24.gruposio.es/ktn/video-fenerbahce-v-alanyaspor-v-tr-tr-1ews-14.php

http://go.acaps.cat/vac/videos-Fenerbahce-Alanyaspor-v-en-gb-1icy30122020-.php

http://live-stream.munich.es/twr/videos-Fenerbahce-Alanyaspor-v-en-gb-1dhg-.php

http://news24.gruposio.es/ktn/video-Fenerbahce-Alanyaspor-v-en-gb-1vpu-.php

http://live07.colomboserboli.com/niy/v-ideos-Fenerbahce-Alanyaspor-v-en-gb-1xks-25.php

http://news24.gruposio.es/ktn/v-ideos-Fenerbahce-Alanyaspor-v-en-gb-1tap-11.php

http://live-stream.munich.es/twr/video-Fenerbahce-Alanyaspor-v-en-gb-1ace30122020-6.php

http://go.acaps.cat/vac/video-Fenerbahce-Alanyaspor-v-en-gb-1msu-9.php

http://news24.gruposio.es/ktn/videos-Fenerbahce-Alanyaspor-v-en-gb-1fyv-3.php

http://go.acaps.cat/vac/videos-Fenerbahce-Alanyaspor-v-en-gb-1msm-9.php

http://live-stream.munich.es/twr/videos-Fenerbahce-Alanyaspor-v-en-gb-1skh-18.php

http://live07.colomboserboli.com/niy/v-ideos-istanbulspor-v-balikesirspor-v-tr-tr-1imi-1.php

http://live07.colomboserboli.com/niy/videos-istanbulspor-v-balikesirspor-v-tr-tr-1ulk-8.php

http://streaming7.actiup.com/kdx/videos-Qatar-SC-Al-Kharaitiyat-v-en-gb-1ube30122020-24.php

ge speakers’ analogy, a few souls use the language for aesthetic purposes and a means of self-expression. It is generally the case for poets and literary writers. How do they look at foreign language words? Well, they’ll undoubtedly see them differently. Their perspective is analogous to my view here, as it looks at language from its power to express ideas while also considering it aesthetically. From this perspective, the use of JSX is unnecessary: it adds noise and does not contribute to the expression of the ideas it expresses.

Another area we needed to improve was our data pipeline testing. This slowed iteration speed and made it difficult for outsiders to safely modify code. We required that pipelines be built with thorough integration tests that run as part of our Continuous Integration processes.

Decorators are wrappers around a function that modify the behavior of the function in a certain way. There are many use cases for decorators, and you may have used them before when working with frameworks like Flask.

As we set out to rebuild our data warehouse, it was clear that we needed a mechanism to ensure cohesion between data models and maintain a high quality bar across teams. We also needed a better way to surface our most trustworthy datasets to end users. To accomplish this, we launched the Midas certification process (depicted in the diagram below).

That’s nice, but you probably already knew these things. What you might not know is that you can also force keyword arguments. The details are described in PEP 3202, but it comes down to using an asterisk before the arguments you want to force as keyword arguments. Or, before everything, forcing all arguments to be keyword arguments:

Last, but not least, we created new mechanisms for ensuring accountability related to data quality. We refreshed our process for reporting data quality bugs, and created a weekly Bug Review meeting for discussing high priority bugs and aligning on corrective actions. We also require that teams incorporate data pipeline SLAs into their quarterly OKR planning.

Data operations was another opportunity for improvement, so we made sure to set strict requirements in this area. All important datasets are required to have an SLA for landing times, and pipelines are required to be configured with Pager Duty.

At this point in time, the Data Quality initiative is moving at full steam, but there is still plenty of work to be done. We’re accelerating investments into our data foundation, designing our next generation of data engineering tools and workflows, and developing a strategy that will shift our data warehouse from a daily batch paradigm to near real-time. We are aggressively hiring data engineering leaders who will develop these architectures and drive them to completion. If you want to help us achieve these goals, check out the Airbnb Careers page.

We also built new tooling for executing data quality checks and anomaly detection, and required their use in new pipelines. Anomaly detection in particular has been highly successful in preventing quality issues in our new pipelines.