How to build an ASP.NET Core API endpoint for time series anomaly detection, particularly spike detection, using ML.NET to identify interesting intraday stock price points.
Background
What is Anomaly Detection?
According to Wikipedia, “In data analysis, anomaly detection (also outlier detection) is the identification of rare items, events or observations which raise suspicions by differing significantly from the majority of the data.”
In plain English, anomaly detection identifies weird behaviors or data points, such as power outages, email fraud, viral tweets, or BIG stock price movements. There are two common types of time series anomalies: spikes and change points.
What is Spike Detection? Spikes are literally what it says, “spikes”… Spikes detection identifies temporary bursts of weird behavior. For example, if the Tesla stock price sky-rockets after they release their quarterly financial report, but then quickly sells off, the peak point is a spike. …
How to deploy a trained sentiment analysis machine learning model to a REST API using Microsoft ML.NET and ASP.NET Core, in just 15 mins.
Why fast integration?
If you have a pure Data Scientist role, you probably care more about studying the data, going through feature engineering, choosing the correct statistical or machine learning model, and training the model to perform well on both the training dataset and validation dataset. And that’s it!
However, in most cases, a trained machine learning model does not just stop there and sit in a Jupyter Notebook, but it needs to be deployed as a service so it can be actually used in production applications. …
Discussing a GitHub starter project to build a web API using Partitioned Repository Pattern with Azure Cosmos DB, ASP.NET Core, and Clean Architecture.
This article describes the GitHub project that can be used as a starting point to work with:
- Clean Architecture (Onion Architecture)
- ASP.NET Core 3.1
- Azure Cosmos DB .NET SDK V3
- Repository Design Pattern
- Partition Key and Partitioned Repository
Popular features also supported in the project include:
- Cosmos DB database initial creation and data seeding
- Query data using Parameterized Query, LINQ and IQueryable, Specification Pattern in Cosmos DB
- REST API with OData support and Swagger UI
- MediatR Command/Query pattern
- MediatR pipeline behaviour for exception handling
- FluentValidation for validation
Please see a full updated feature list in the GitHub repo. If you want to dive straight into the code, please see GitHub repository Clean Architecture with partitioned repository pattern using Azure Cosmos DB. The GitHub repository is work in progress. …
Discussing an All-in-One starter project for Azure Functions to work with Cosmos DB using Partitioned Repository Pattern, with features like Dependency Injection, structured logging, strongly-typed configuration, email service, etc..
Goal Overview
The goal of this article is to discuss a starter project that can be used to work with Azure Functions and Cosmos DB and be (almost) production ready without having to worry about solution design. This project is designed in Clean Architecture, and includes many of the common features that you would expect on a production environment, such as structured logging using Serilog, strongly-typed configuration, Dependency Injection for Inversion of Control, email service, etc..
Before we dive into the actual project, let’s do a quick overview.
Azure Cosmos DB — 10,000-Foot Overview
Azure Cosmos DB is the managed NoSQL database service provided by Microsoft. It supports APIs like SQL, Cassandra, and MongoDB etc., which allows the developers to use query language of their choice. The most appealing feature to me myself as a Data Engineer is that Cosmos DB takes care…
Discussing how to scale the barebones TinyURL system using Load Balancer and Database Horizontal Partitioning, and related topics like partition key design strategy, partition vs replica vs backup, SQL vs NoSQL.
In the last couple of articles, we discussed how to design a barebones working solution for TinyURL, and also how to improve 90% of the estimated workload by introducing cache.
The system diagram below shows what we currently have. The single web server handles all the HTTP requests from the clients, and the single database stores all the data and handles all the read and write requests. Let’s trace the workflow and see how we can scale each component in the diagram, starting from the web server.
Discussing how to use Serilog for both filesystem and log stream on Azure App Service for ASP.NET Core Web API in details.
Goal
Does this sound familiar? “XXX works perfectly locally, but does not work at all in production”. The same mystery applies to structured logging using Serilog!
This is a very short article aiming to discuss how to setup Serilog in an ASP.NET Core Web API project so that it works flawlessly both in development locally and also in production on Azure App Service.
Getting Started
Not to waste any time, so let’s get straight into it!
Prerequisites:
- A new or existing ASP.NET Core API project
- (optional) API deployed on Azure App Service if you want to test production environment
Step 1 — Add Environment…
Discussing how to automatically create an Azure Cosmos DB database and its containers on API application startup in Clean Architecture using ASP.NET Core.
Background
It is straightforward to create Azure database and containers in Azure Portal. But if you are developing a new application using Cosmos DB Emulator, it will be nice to be able to delete the whole database and have it automatically re-created on application start to get a clean state. Also, if a new developer joins the team, he/she can just run the application and be all set to go, instead of having to manually create a database and containers (annoying!).
Prerequisites
- (optional) Cosmos DB Container Factory class library, if you want to just copy the code and be ready to get going. Refer to my previous article, Best Design Pattern for Azure Cosmos DB Containers — Factory Pattern, on how to implement Factory Pattern for Cosmos DB Containers. …
Discussing how to build a powerful factory to work with Cosmos DB containers and encapsulate the complex logic with Cosmos DB .NET SDK in Clean Architecture using ASP.NET Core.
Background
What is Factory Pattern? It is a design pattern in OOP that allows us to encapsulate object creation logic and hide its complexity from the calling program, such as API, Function Apps, and even users. This probably sounds very vague. My personal take-away note is that factory pattern allows me to ask the factory:
- to make me a car, then I will get a car
- to make me a train, then I will get a train
- to make me a universe, then I will get a universe
I do not have to worry about how to make a car/train/universe throughout the whole process, which successfully decouples the “how-to” from me, the calling program. As is in the image above, I only need to know the factory produces Cosmos DB containers, but do not have to know what’s inside the factory. …
Discussing how exceptions can be handled in a centralized spot and how to return consistent responses. Exception logging is also centralized in the same approach.
Motivation
I adopted a REST API project today which had no exception handling or logging, making it really hard to investigate issues. So I spent an hour adding a centralized exception handler and structured logging using Serilog. In this article, I’d like to discuss how exception handling and logging can be setup at the beginning phase of a project following a best and easy concept.
Background
Once the business-level entities are defined, common exceptions and business-specific exceptions should be defined so that they can be shared among projects like API, Function App, Console App, etc..
Once the exceptions are defined, we need a centralized spot to handle all the pre-defined exceptions as well as the unknown exceptions (most likely a 500 error), log the errors, and then return a consistent API response so that any client that calls the API knows what to expect. …
Discuss a class library that can be used for design-time DbContext creation and database creation for ASP.NET Core Identity using EF Code First, without a web or API project.
Goal
The goal is to create a stand-alone class library whose sole purpose is to host ASP.NET Core Identity models, DbContext and EF Core migrations, and be able to create Identity database by applying migrations without relying on an actual web host. This is particularly useful if you follow Microsoft recommended Clean Architecture and want to develop your core entities and infrastructure first and not worry about the web app or the API (UI layer).
This article extends the Microsoft documentation on how to create DbContext from a design-time factory.
See screenshot below for the folder structure of the class library, “CleanArchitectureCosmosDB.Infrastructure”, that fully supports ASP.NET Core Identity DbContext creation, customized identity models like ApplicationUser which inherits IdentityUser, and migrations management, and database creation through applying migrations. …
About
