Text-to-Video Model Benchmark

• Project code  
• Report  
• Poster  

T2V-Bench is a fully automated method for evaluating Text-to-Video (T2V) generative models in both coarse and fine-grained, object-oriented manner.

The method comprises five key task areas were T2V models were identified to typically underperform, namely: object identification, object counting, color identification, color attribution, and general video consistency. T2V-Bench employs a two-stage pipeline including evaluation of five aforementioned model generation capabilities, focusing on perception and reasoning.

Stage 1

In the first stage, perception is evaluated by verifying whether the generated videos satisfy the semantic requirements specified in the initial prompts used to create them. That is achieved through video captioning and caption-prompt alignment assessment. Videos generated by the T2V model are captioned using an video-to-text model. Subsequently, these generated captions are compared to the original prompts using an LLM to determine their mutual similarity.

Stage 2

The second stage assesses the T2V model’s reasoning capabilities, including object count and color identification as well as attribution, on a frame-based level. Leveraging four specific tasks: object detection and counting, color identification and attribution, this stage examines if specific video features, such as object color and quantity, align with the prompts initially provided to the model.

ADAŚ - Autonomous Vehicle

• Project Website  
• Demo Video  
• Simulator Video  

"ADAŚ is an autonomous RC car equipped with Advanced Driver Assistance Systems (ADAS). This vehicle utilizes AI-based perception and driving systems developed by the Autonomous Driving Team at the ADek Student Research Association. As the team leader, I managed a group of seven developers.

Together, we designed the architecture for the autonomous driving system from the ground up and developed advanced vision-based driver assistance systems. We then implemented and combined them using Robot Operating System (ROS).

ADAŚ in person

ADAŚ is capable of detecting various objects on the road, such as other vehicles, pedestrians, and road signs. It can also recognize traffic lights and road lanes, and it can navigate in the environment using a camera and a LiDAR sensor. You can see ADAŚ below:

Simulator

In the video below, you can see ADAŚ in action in a simulator environment, which our team created to test the vehicle's capabilities in a controlled environment. Notably, ADAŚ can detect other road users and objects, follow lanes, as well as recognize both traffic signs and lights and respond accordingly.

Real-life data

Our autonomous driving system has been tested in real-life conditions, where it has demonstrated its promising capabilities in detecting and recognizing various objects on the road, as well as planning driving maneuvers accordingly. See the video below for a demonstration our system in action.

Customer Service AI Trainer

• Project Code  
• Demo Video  

This following project was created as a startup idea for an AI-based chat platform for training Customer Service agents. The entire app is AI-powered and uses the OpenAI chat completion API to simulate a potential customer as well as to provide feedback to the agent.

Demo Video

Agents may respond to the AI-generated customer queries, and the platform will provide them with immediate, real-time feedback, which they can use to improve their responses.

System Architecture

The platform was built using Python, FastAPI, and ReactJS and was containerized using Docker. Moreover, I have created an OpenAI chat completion API wrapper to facilitate the communication between the platform and the OpenAI API. The system architecture is presented in the diagram below.

Knowledge Distillation Library

• Project Code  
• Demo Video  

A project created for my Bachelor's Thesis - "Tuning Small Semantic Segmentation Models via Knowledge Distillation". The idea was to create a small and robust Semantic Segmentation model, for the purpose of segmenting road scenes in a fully autonomous vehicle. Hence, while preserving good performance the model has to work in real time.

To achive both, a technique known as Knowledge Distillation (KD) was employed in order to transfer knowledge from a large and complex SegFormer B5 model, to a small SegFormer B0 model, being lightweight enough to work on most devices in real time. To this end, I created a framework for Knowledge Distillation, that is based on PyTorch and works as an add-on to the popular HuggingFace Transformers library. The framework is designed to be easily extendable and customizable, so that it can be used for various tasks and models.

Training Pipeline

The model has been trained on the Mapillary Vistas dataset, which is the most comprehensive and diverse, publicly available road scene segmentation dataset, comprising 25 000 annotated samples.

Furthermore, a Response-Based Knowledge Distillation approach is employed for training the SegFormer model. The training pipeline is presented in the diagram below.

Results

A demonstration video of the trained model in action is presented below.

By introducing the KD learning scheme, the performance of SegFormer B0 increased by approximately 2.5% while preserving its previous compact size. While the applied knowledge distillation method boosts the model’s performance for every class, as depicted in the figure below, the leveraged technique appears to be favorable for the minority classes, i.e. "Train" and "Bicycle".

Predicting the Number of Returned Products

• Project Code  

Predicting the number of shop returns using time series and regression. A university project that comprised a complete business problem analysis as well as data preparation and processing. This was ultimately complemented by training a regression model and deploying it with Flask and Docker.

Context: An online computer shop called "eSzoppping" is our client. The shop warehouse workers are struggling with organising and processing order returns, as they don't know how many packages to expect.

Task: The goal was to create a model, which will support the warehouse workers by predicting the number of returned orders for the next 14 days.

Approach

The model was trained on the dataset containing the number of returned orders for the last 3 years. I conducted a thorough explanatory data analysis (EDA) of the dataset, which allowed me to identify patterns in the data as well as the most important features for the model and to preprocess the data accordingly.

Results

Seeking the best model, I experimented with numerous regression models, including Linear Regression, Gradient Boosting, and Random Forest. The latter model achieved the best results, with a Mean Absolute Error (MAE) of 1.015 suggesting that it was able to predict the number of returned products for each day with an error of 1.015 units.
See the results in the figure below and in the first figure above.

Ultimately, the fine-tuned model was deployed using Flask and Docker, which allowed the client to access the model via a simple web interface created in React.

Generating Traffic Signs with Generative Networks

• Project Code  

The following project was created as part of the Deep Learning course at the Warsaw University of Technology. It involved generating synthetic traffic sign image data using Generative Networks in PyTorch.

To this end, I built and experimented with various variations of VAE namely:
- Vanilla VAE
- Conv VAE

as well as GAN:
- DCGAN
- VAEGAN

The models were trained on the German Traffic Sign Recognition Benchmark (GTSRB) dataset, which contains over 50,000 images of 43 different traffic signs. To evaluate them, I used the Frechet Inception Distance (FID) metric, which measures the similarity between the generated and real data distributions.
See the results of the generations in the image above.

Reconstructions

The models were also tested on their ability to reconstruct the input images in order to verify their latent space representation. The results of the reconstructions are presented in the image below.