What do you need to know to start your startup using artificial intelligence technologies? What processes do you need to consider, how do you need to prepare and process data for training a neural network, and what to look for when it comes to recruiting and testing? All this must be studied before fraternizing for work.
Understanding both the business needs and the source data sources is critical. It happens that a good solution appears without the use of artificial intelligence technologies.
What do you need to focus on to make the right decision, on the needs of the users, or the technology being used? It is believed that such a product using AI, which is designed to help people access information or directly process such information, to make the right decisions, is successful.
1. Determining the goal and setting the task
The first and most important question for the development team is the correct formulation of the problem, the solution of which through artificial intelligence will be a business need. As a result of solving this problem, the final expected result will be obtained, and the result must be checked to ensure maximum accuracy.
2. Collection and preparation of initial data
The next step is to decide on the data sources. It is necessary to make sure how true they are, what are the ways of integrating this data into the system being developed, and how the development team can use it to achieve business goals. Simply put, when working with data, it must be processed as efficiently as possible, despite complex transformations and low processing speed, to get the correct result at the output. The quality of the primary data does not matter, the main thing is to properly prepare the training sample. If the principle of training data sampling is chosen correctly, then the task can be scaled using a more complete data set.
Summarizing what has been said, you can prepare a list of questions:
• What data sources are available?
• What is this data?
• Do we have enough data?
• Where does the data come from, how is the data collected and processed?
Several questions should also be asked in the future, since if this is not done, then this fact can significantly complicate the development of the project.
Additional questions:
✓ Is the sample of data we use to train the model representative?
✓ Are there any obstacles that we did not consider?
✓ Does the sample contain personal data and are we entitled to use it?
The next step that must be taken before creating the algorithm itself is processing. This implies by itself (preparation, clustering) and data normalization. This step is needed to prepare the sample for correct interpretation by the mathematical model of artificial intelligence. For example, operations such as raising variables to power or multiplying by a constant can be performed on a numeric selection. This will allow linear models to model non-linear relationships to identify common patterns.
Sometimes it is necessary to perform a transformation to correctly interpret the frequency characteristics in audio processing or to use the SIFT algorithm when solving a problem of image matching. You need to understand that data normalization and preparation is critical to traditional machine learning. This process significantly affects the choice of the architecture of the neural networks used, especially in the so-called deep learning, when it is necessary to correctly determine the number of hidden layers in the neural network and the number of artificial neurons in them. One of the main advantages of multilayer neural networks is modeling the work of a certain complex mathematical relationship.
3. Development of an artificial intelligence model, programming, and training of a neural network
Now that there is a clear business goal, the correct set of initial data for the sample, and the sample itself, you can start developing neural network models, programming, and further training the neural network. The new stage will include the choice of the learning algorithm, the application of the learning algorithm, its visualization, and the assessment of the quality of learning.
Training a neural network can be compared to training a dog a command, done millions of times. As trivial as it sounds, the learning process is pretty simple. You have a training sample containing inputs and outputs. You feed the initial data to the input of the neural network and at the output, you get a certain result of processing by the neural network. Next, you compare the result with the final result from the above sample, indicating the degree of similarity. The process may seem simple, but its efficient and correct execution on large data sets is far from easy to ensure. It is necessary to choose the right learning algorithm for the neural network, otherwise, the created artificial intelligence can learn to misinterpret the incoming data stream, which will lead to unwanted errors.
One such case happened with Google when a photo of an African American family was loaded into its facial recognition software, and the program tagged the image as a family of monkeys. Is the result of the work of AI-related to racism in this case? As a result, it turns out that the final behavior of the created artificial intelligence follows from the set of initial data, the procedures for their processing and normalization, and the training algorithm used, and the criterion for validating the training result. It is at this stage that the combination of several approaches makes it possible to properly train the neural network so that interactions with the developed mathematical model are as effective as possible. One of the defining decisions at this stage will be what percentage of the training sample will be entirely used for training the neural network, and what percentage will be presented to the neural network later for subsequent testing.
4. Verification of the obtained results
The sample for testing should also be representative, as well as the dataset for training. Previously, the rule of thumb was to use the original training set in an 80 by 20 ratio, of which most of the data is used to train the neural network. Now it remains to decide on the composition of the team and the role of each participant. The development team should take over the work of training the neural network, in particular the development and testing of machine learning algorithms. One part of the team will conduct training, and the other will test the algorithm, checking how accurately the AI solves the problem. One of the common mistakes that many development teams face is using the same dataset to train and test a neural network. This usually leads to the fact that the technology will show very good results during testing since the test data was presented at the training stage.
A development team that falls into this trap will be tempted to say that as a result of the machine learning carried out, the neural network gives the correct result in the output. Next, the development team must evaluate the performance of the neural network model in terms of sensitivity and cost. There are no perfect models, but not all errors are the same on their own. The model can produce a so-called false-positive or false-negative result. Of course, a false positive is better. It makes it possible to play it safe a little when making a decision.
In the case of retail solutions, over-forecasting can increase working capital costs, while underestimation can lead to more significant costs, due to a sharp decline in customer sales. Deciding how to properly train a neural network to solve a specific problem is always a trade-off.
How to answer the question, how many training cycles are needed?
Is the work of AI just memorizing an array of data and retrieving it, or will the priority be identifying certain patterns for the data in this array?
Balancing neural network overfitting and generalization is another tricky science. Throughout the entire learning process, the main problem often remains, as it were, hidden, disappears from the field of visibility. For example, how to measure success, value, preference? Is this measure consistent with business requirements? Is this technically helpful? These questions often need to be answered right away. All datasets have weaknesses and pitfalls, and not all are mathematically useful for learning.
Once the development team is confident in the correct solution to the problem, they can work on transforming the results of the neural network into ideas, action elements, predictions, or simply use them as a result of data processing.