Creating a tool for applying decals is fairly useful…
I consider the case in which a decal is a single planar polygon with a texture (be it transparent, cutout or neither)
An artist would want to take the polygon (with texture) and parent it to the camera, then the artist can move around until a nice spot for the decal is found. In the meantime the decal can be panned and rotated (only around the camera forward axis) so it always remains in the same plane as the viewport.
Once the decal looks nice from the point of view of the camera, it needs to be applied in 3D and separated from the camera.
This initially is a straightforward action: for each vertex we need to shoot a ray from the camera position, through the vertex, onto our world or target mesh. The intersection point minus a little offset (to avoid coplanar faces which cause flickering) is where the vertex should be set.
Problems arise however when applying the decal on a curved surface, it will be floating in front of or penetrating through the surface…
This can be solved in another way. Fastest is if we know what mesh we wish to apply our decal on, otherwise we’d need to combine the world mesh for this and that gets heavy to compute really fast.
Transform the mesh to world space, then transform it to camera space (or the decal’s object space).
Now divide x,y by z to put the mesh onto the view plane, from this point on, our case is 2D.
Foreach edge in the mesh: check intersection with each edge in the decal polygon.
If they intersect, insert a vertex into the polygon (or schedule the insertion until all checks are performed).
When done inserting vertices (which in a simple single non-triangulated polygon case is easily accomplished by inserting a point at the right place in the list).
We can first merge the vertices with a small tolerance in case we graze the tip of a triangle and split redundantly much at some point.
Here we have an original vertices and three splits very close to eachother merging these will make the result cleaner and have less ugly normals.
Next things to do:
Raycast all the vertices in world space onto the mesh in world space as described before
Calculate normals (cross product 2 edges of each triangle: (p2-p0) X (p1-p0))
Unparent from the camera (if you wrote a tool that did that for the artist)
Note that it is possible to flatten the decal polygon in camera space as well, making that it does not need to be parented to the camera and a user is free to modify it in any way.