BloomTwins XR

Tao et al. mention precisely what we were aiming for in our experience and taking into consideration during our brainstorming: "AR (including MR) contains a set of innovative techniques (e.g., real-time data acquisition, human-computer interaction, scene capture, real-time tracking, and registration, etc.), and can augment the view of the physical world by embedding computer-generated elements or objects. It enables people to work in a physical environment with valuable and abundant virtual information and models". Therefore, after creating a possible scenario, we decided on interactions that included manipulating the physical plant and its digital counterpart, engaging with data, and fostering user-to-user interaction. The plant's 3D model was designed using the 3D modeling software Blender. During modeling, we focused on individual leaves' number, size, shape, and placement on our physical template. We then imported the 3D model into Unity and created the virtual environment components for visualization. Photon is a real-time multiplayer game development framework that allows for collaborative experiences. We used it to implement a real-time collaborative experience. We started designing our data visualization in Figma to get a first idea and later designed a similar dynamic visualization in Unity. It is important to emphasize that we used simulated data for this project.

Although some technologies, such as connected pots and sensors capable of measuring nutritional values and environmental conditions, were unavailable, we recognized their potential value for future project iterations. For instance, the Xiaomi Mi Flora sensor, which measures light level (lux), temperature, humidity, and nutrient levels in the soil, could have provided valuable real-time data.

We collected data from January 1, 2020, and May 31, 2023, to gather historical weather data for Stockholm. This data served as a foundation for modeling the environmental conditions experienced by the plant. To showcase historical data in the MR experience, we developed a script in Unity that reads data from a CSV file and updates sliders representing temperature, light, and humidity. This feature enabled users to observe the environmental and soil conditions over the past seven days. Regarding plant growth, we collected data on the monthly growth in centimeters and the number of leaves produced for minimal and maximal optimal conditions. We also determined the minimum and maximum values for temperature, humidity, light, pH, nitrogen, phosphorus, and potassium. We employed the logistic growth equation, a commonly used mathematical model for describing population growth to estimate plant growth. In our case, this equation was applied to approximate the growth of the Ficus elastica plant. Thelogistic growth equation used in the script is: N(t) = K / (1 + (K / N0 - 1) * exp(-r * t)). The script solved it for time(t), utilizing the given values of N0 (initial leaves), N(t) (target leaves), K (carrying capacity), and r (growth rate). By inputting these values, the script calculated the estimated time required for the Ficus elastica plant to grow from the initial number of leaves to the target number of leaves. An estimated growth time of months was then displayed in the Unity console.

Given the novelty and limited testing of MR technologies, we had considerable difficulty obtaining guidance, instructions, and inspiration on integrating the various components. As a result, we spent much time exploring and testing different approaches. However, since this is a typical challenge faced with new technologies, this exploratory approach is one of the goals of this course. One challenge we encountered was figuring out how to handle and implement the collaborative part the way we needed. The synchronization of objects and interfaces in the multiplayer space between the two users participating in the experience was a significant challenge, as the positions and states of objects in the virtual world were not always behaving the way we intended. BloomTwins uses the "Passthrough" functionality of Meta Quest Pro headsets, enabling the user to see the surrounding environment, which is necessary to implement an MR application. This presented another challenge, as many of the frameworks we tried to implement in multiplayer did not support Passthrough. This meant we had to spend considerable time exploring possible technologies for the project.