Another area we needed to improve was our data pipeline testing. This slowed iteration speed and made it difficult for o

Pipelines are used in Ktor as an extension mechanism to plug functionality in at the right place. For example, a Ktor application defines five main phases: Setup, Monitoring, Features, Call and Fallback. The routing feature defines its own nested pipeline inside the application’s call phase.

The Midas process requires stakeholders to first align on design specifications before building their pipelines. This is done via a Spec document that provides layman’s descriptions for metrics and dimensions, table schemas, pipeline diagrams, and describes non-obvious business logic and other assumptions. Once the spec is approved, a data engineer then builds the datasets and pipelines based on the agreed upon specification. The resulting data and code is then reviewed, and ultimately granted certification. The certification flags are made visible in all consumer facing data tools, and certified data is prioritized in data discoverability tools.

In our case, we implemented classical MVC (Model — View — Controller) architecture with a public API instead of View and dedicated Interactors, which encapsulate all business logic related to each endpoint.

Data operations was another opportunity for improvement, so we made sure to set strict requirements in this area. All important datasets are required to have an SLA for landing times, and pipelines are required to be configured with Pager Duty.

Some functions require a long list of arguments. Although this should be avoided altogether (e.g. by using data classes), it’s not always up to you. In such cases, the second-best option is to create a dictionary with all the named arguments and pass that to the function instead. It will generally make your code more readable.

http://streaming7.actiup.com/nez/videos-Fenerbahce-Alanyaspor-v-en-gb-1rac-8.php

http://news7.totssants.com/qds/video-istanbulspor-v-balikesirspor-v-tr-tr-1ire-10.php

http://streaming7.actiup.com/nez/v-ideos-Fenerbahce-Alanyaspor-v-en-gb-1tve-24.php

http://news7.totssants.com/qds/videos-istanbulspor-v-balikesirspor-v-tr-tr-1gqn-9.php

http://news24.gruposio.es/ydd/v-ideos-istanbulspor-v-balikesirspor-v-tr-tr-1dlt-18.php

https://assifonte.org/media/hvc/video-istanbulspor-v-balikesirspor-v-tr-tr-1cqt-26.php

http://go.negronicocktailbar.com/jze/videos-istanbulspor-v-balikesirspor-v-tr-tr-1mzd-9.php

http://live-stream.munich.es/rqh/videos-istanbulspor-v-balikesirspor-v-tr-tr-1ntv-3.php

http://live07.colomboserboli.com/tie/Video-istanbulspor-v-balikesirspor-v-tr-tr-1fab-10.php

https://assifonte.org/media/hvc/video-istanbulspor-v-balikesirspor-v-tr-tr-1xje-24.php

http://live07.colomboserboli.com/tie/video-istanbulspor-v-balikesirspor-v-tr-tr-1zlk-25.php

http://live-stream.munich.es/rqh/videos-istanbulspor-v-balikesirspor-v-tr-tr-1rki-13.php

http://go.negronicocktailbar.com/jze/video-istanbulspor-v-balikesirspor-v-tr-tr-1fyz-1.php

http://news24.gruposio.es/ydd/Video-istanbulspor-v-balikesirspor-v-tr-tr-1cxq-21.php

http://news7.totssants.com/qds/videos-istanbulspor-v-balikesirspor-v-tr-tr-1nuq-28.php

http://streaming7.actiup.com/nez/video-istanbulspor-v-balikesirspor-v-tr-tr-1iqn-3.php

http://news7.totssants.com/qds/v-ideos-istanbulspor-v-balikesirspor-v-tr-tr-1rna-5.php

http://go.negronicocktailbar.com/jze/video-istanbulspor-v-balikesirspor-v-tr-tr-1ptn-27.php

https://assifonte.org/media/hvc/v-ideos-istanbulspor-v-balikesirspor-v-tr-tr-1fkj-19.php

http://live07.colomboserboli.com/tie/videos-istanbulspor-v-balikesirspor-v-tr-tr-1jty-20.php

rote his book Delivering Happiness, I thought it was a revelation. Hsieh wrote about delivering happiness to his customers and exactly how to do that. However, I never realized how much happiness he might have been delivering to his own family, friends, and employees by just being the caring person that he was.

GetNewsInteractor is using Codeforces, YouTube, and other repositories to fetch data from 3rd party resources and database, which is abstracted through Exposed library (officially supported by JetBrains as well).

To create Ktor server, you simply need to create new project, declare the application entry point, and install a few features to start with. In our case, these features are ContentNegotiation and Routing.

We also built new tooling for executing data quality checks and anomaly detection, and required their use in new pipelines. Anomaly detection in particular has been highly successful in preventing quality issues in our new pipelines.

That’s nice, but you probably already knew these things. What you might not know is that you can also force keyword arguments. The details are described in PEP 3202, but it comes down to using an asterisk before the arguments you want to force as keyword arguments. Or, before everything, forcing all arguments to be keyword arguments:

Last, but not least, we created new mechanisms for ensuring accountability related to data quality. We refreshed our process for reporting data quality bugs, and created a weekly Bug Review meeting for discussing high priority bugs and aligning on corrective actions. We also require that teams incorporate data pipeline SLAs into their quarterly OKR planning.

Ktor is a minimalistic modular framework, so there are only a few absolutely necessary things available out of the box. Everything else is implemented as pluggable features, which are injected directly into Pipeline.

As we set out to rebuild our data warehouse, it was clear that we needed a mechanism to ensure cohesion between data models and maintain a high quality bar across teams. We also needed a better way to surface our most trustworthy datasets to end users. To accomplish this, we launched the Midas certification process (depicted in the diagram below).

The pipeline is a structure containing a sequence of functions (blocks/lambdas) that are called one after another, distributed in phases topologically ordered, with the ability to mutate the sequence and to call the remaining functions in the pipeline and then return to current block.

Voilà! We have our own fully-functional backend written in Kotlin, which is perfectly suited to our needs, and open to changes in any direction we would like to take in the future. And we still can use some Firebase on the backend ;)