react-native-vision-camera-mlkit

📦 A Google ML Kit frame processor plugin for react-native-vision-camera.

This plugin provides a simple way to use various ML Kit Vision APIs in your React Native App's Frame processor.

⚠️ This plugin is still in development and not yet ready for iOS.

🧵 Vision APIs Roadmap

This roadmap provides an overview of the progress for implementing various ML Kit Vision APIs. Note that all completed items are currently only available for Android. iOS support is still in development.

#	APIs	Android	iOS
1	Barcode scanning
2	Face detection
3	Face mesh detection
4	Text recognition v2
5	Image labeling
6	Object detection and tracking
7	Digital ink recognition
8	Pose detection
9	Selfie segmentation
10	Subject segmentation
11	Document scanner

💬 Video and image analysis APIs to label images and detect barcodes, text, faces, and objects.

Barcode scanning

Scan and process barcodes. Supports most standard 1D and 2D formats.

Face detection

Detect faces and facial landmarks.

Face mesh detection

Detect face mesh info on close-range images.

Text recognition v2

Recognize and extract text from images.

Image labeling

Identify objects, locations, activities, animal species, products, and more. Use a general-purpose base model or tailor to your use case with a custom TensorFlow Lite model.

Object detection and tracking

Localize and track in real time one or more objects in the live camera feed.

Digital ink recognition

Recognizes handwritten text and handdrawn shapes on a digital surface, such as a touch screen. Recognizes 300+ languages, emojis and basic shapes.

Pose detection

Detect the position of the human body in real time.

Selfie segmentation

Separate the background from users within a scene and focus on what matters.

Subject segmentation

Separate subjects (people, pets, or objects) from the background in a picture.

Document scanner

Digitize physical documents from pictures.

🚀 Getting Started

🚨 Required Dependencies

Ensure you have installed the required packages before installing this plugin.

#	Package	Version
1	react-native-vision-camera	>=4.0.1
2	react-native-worklets-core	>=1.2.0

Follow the installation instructions for each package.

💻 Installation

To install the plugin, run:

npm install react-native-vision-camera-mlkit
# or
yarn add react-native-vision-camera-mlkit

🪝 Hooks

useBarcodeScanner (Barcode scanning)

  const { barcodeScanner } = useBarcodeScanner();

  const frameProcessor = useFrameProcessor(
    (frame) => {
      'worklet';

      runAsync(frame, () => {
        'worklet';

        const barcodes = barcodeScanner(frame);
        console.log(barcodes);
      });
    },
    [barcodeScanner]
  );

useImageLabeler (Image labeling)

  const { imageLabeler } = useImageLabeler();

  const frameProcessor = useFrameProcessor(
    (frame) => {
      'worklet';

      runAsync(frame, () => {
        'worklet';

        const labels = imageLabeler(frame);
        console.log(labels);
      });
    },
    [imageLabeler]
  );

useObjectDetector (Object detection and tracking)

  const { objectDetector } = useObjectDetector();

  const frameProcessor = useFrameProcessor(
    (frame) => {
      'worklet';

      runAsync(frame, () => {
        'worklet';

        const objects = objectDetector(frame);
        console.log(objects);
      });
    },
    [objectDetector]
  );

useTextRecognizer (Text recognition v2)

  const { textRecognizer } = useTextRecognizer();

  const frameProcessor = useFrameProcessor(
    (frame) => {
      'worklet';

      runAsync(frame, () => {
        'worklet';

        const text = textRecognizer(frame);
        console.log(text);
      });
    },
    [textRecognizer]
  );