Discussing how exceptions can be handled in a centralized spot and how to return consistent responses. Exception logging is also centralized in the same approach.
Motivation
I adopted a REST API project today which had no exception handling or logging, making it really hard to investigate issues. So I spent an hour adding a centralized exception handler and structured logging using Serilog. In this article, I’d like to discuss how exception handling and logging can be setup at the beginning phase of a project following a best and easy concept.
Background
Once the business-level entities are defined, common exceptions and business-specific exceptions should be defined so that they can be shared among projects like API, Function App, Console App, etc..
Once the exceptions are defined, we need…
Discussing a GitHub starter project to build a web API using Partitioned Repository Pattern with Azure Cosmos DB, ASP.NET Core, and Clean Architecture.
This article describes the GitHub project that can be used as a starting point to work with:
- Clean Architecture (Onion Architecture)
- ASP.NET Core 3.1
- Azure Cosmos DB .NET SDK V3
- Repository Design Pattern
- Partition Key and Partitioned Repository
Popular features also supported in the project include:
- Cosmos DB database initial creation and data seeding
- Query data using Parameterized Query, LINQ and IQueryable, Specification Pattern in Cosmos DB
- REST API with OData support and Swagger UI
- MediatR Command/Query pattern
- MediatR pipeline behaviour for exception handling
- FluentValidation for validation
Please see a full updated feature list in the GitHub repo. If you…
How to build an ASP.NET Core API endpoint for time series anomaly detection, particularly spike detection, using ML.NET to identify interesting intraday stock price points.
Background
What is Anomaly Detection?
According to Wikipedia, “In data analysis, anomaly detection (also outlier detection) is the identification of rare items, events or observations which raise suspicions by differing significantly from the majority of the data.”
In plain English, anomaly detection identifies weird behaviors or data points, such as power outages, email fraud, viral tweets, or BIG stock price movements. There are two common types of time series anomalies: spikes and change points.
What is Spike Detection? Spikes are literally what it says, “spikes”… Spikes detection identifies temporary bursts of weird behavior. For example, if the Tesla stock price sky-rockets after they release their…
How to deploy a trained sentiment analysis machine learning model to a REST API using Microsoft ML.NET and ASP.NET Core, in just 15 mins.
Why fast integration?
If you have a pure Data Scientist role, you probably care more about studying the data, going through feature engineering, choosing the correct statistical or machine learning model, and training the model to perform well on both the training dataset and validation dataset. And that’s it!
However, in most cases, a trained machine learning model does not just stop there and sit in a Jupyter Notebook, but it needs to be deployed as a service so it can be actually used in production applications. …
Discussing how to scale the barebones TinyURL system using Load Balancer and Database Horizontal Partitioning, and related topics like partition key design strategy, partition vs replica vs backup, SQL vs NoSQL.
In the last couple of articles, we discussed how to design a barebones working solution for TinyURL, and also how to improve 90% of the estimated workload by introducing cache.
The system diagram below shows what we currently have. The single web server handles all the HTTP requests from the clients, and the single database stores all the data and handles all the read and write requests. Let’s trace the workflow and see how we can scale each component in the diagram, starting from the web server.
Discuss how to add a message producer to a distributed messaging system using REST API with ASP.NET Core and publish messages to the message broker using RabbitMQ.
Goal
In a previous article, we built a distributed messaging system where a message broker using RabbitMQ and message consumers using ASP.NET Core hosted services were set up and running. In this article, we will discuss how to add a message producer using ASP.NET Core REST API to the messaging system, as colored in green in the system diagram above. Once that’s completed, we will demo sending and consuming messages in the message broker.
Prerequisites
If you have not, I would recommend reviewing the previous article to get the message broker and message consumers up and running. …
Get Redis, SQL Server, MongoDB, and more up and running in docker in 15 mins!
Background
Let me start by asking and answering a few quick questions:
- Have you installed a full SQL Server on your dev machine? Yes, I have.
- Have you installed Redis on your dev machine? Yes, I have.
- Have you installed all the other databases that you temporarily need on your dev machine? Yes, I have….
- Do you need them all running in the background on your dev machine, eating all the CPU and Memory, until the world ends? No, I do not…
We often go through tedious installation processes and configuration processes in order to get a database up and running…
Discuss how to quickly setup a distributed messaging system, including consumers and RabbitMQ with Docker in ASP.NET Core, hopefully, in less than an hour.
Background
What is distributed messaging system? A distributed messaging system allows asynchronous processing of messages or workloads in microservices architectures. Such a design decouples the message producers, message consumers and the message transport, allowing the design to be highly scalable. Amazon documentation does a great job explaining the basics. Here is the link.
If you have placed an order on Amazon, you will notice that your order gets submitted almost instantly after validating the payment method. After that, you can walk away and a suite of subsequent events would occur. …
Discussing how to implement an application-specific JWT token service provider using ASP.NET Core Identity in Clean Architecture.
Background
OAuth 2.0 is quite popular and is in-fact adopted almost everywhere now. Even if you haven’t developed any OAuth workflows, I am fairly certain that you have used it. Every time you provide consent to allow a specific application to access data on your behalf from Google, Facebook, or GitHub etc., you are using the OAuth 2.0 for authorization. You likely have used OpenID Connect for authentication as well if you have ever looked closer and noticed any id token. But do all authentication and authorization workflows have to use OAuth 2.0 and OpenID Connect? Not really.
Even though many…
Discussing Binary Search, and how to use its template code with Two Pointers to solve multiple interview questions in C# for a better understanding of Data Structure and Algorithms.
Goal
The goal is to share some concise notes on Binary Search, and its implementation using both recursion and two pointers algorithm. Hopefully, the high-level summary notes allow you to add Binary Search to your tool bag without getting swamped by detailed textbook explanations. The following items are covered:
- Algorithm high-level logic
- Typical use cases/interview questions referencing LeetCode
- Typical bugs/gotchas in implementation
What is Binary Search?
Binary Search is an algorithm to search a sorted array for a target value. Important notes on Binary Search:
- O(logn) time complexity, as the problem size is reduced to half after each iteration
- O(1) space complexity
- works on sorted…
Shawn Shi
Software Engineer, Machine Learning Engineer. When I am not dived into data and code, I am hanging out with my young daughter or outside rock climbing!
